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Teaching Social Skills to Elementary Students ” 1 

The findings from this review do not reflect the full body of research evidence on We Have Skills! 


What is this study about? 

The study authors examined the effects of We Have 
Skills! (WHS), a supplemental, video-based social 
skills program for early elementary students. WHS 
consists of three components: instructional materi- 
als, teacher professional development, and stu- 
dent assessment and was developed by the study 
authors. 

The study sample included 17 elementary schools 
in four school districts located in California, Oregon, 
and Washington. Teachers of grades K-3 at study 
schools were given the opportunity to participate in 
the evaluation. Consenting teachers were then ran- 
domly assigned either to a WHS intervention group 
or to a comparison group. Teachers in the interven- 
tion group received training on the WHS intervention 
and were asked to use the intervention with stu- 
dents in their classrooms. 

A total of 83 classrooms were randomly assigned 
to the intervention and comparison groups, and 70 
classrooms were included in the analytic sample (37 
in the intervention group and 33 in the comparison 
group). From a total of 1 ,661 students at the time of 
random assignment, 1,466 students were included 
in the analytic sample (822 in the intervention group 
and 644 in the comparison group). 


Study authors examined the impact of WHS on 
student social skills using the Elementary School 
Behavioral Assessment (ESBA), which study teach- 
ers used to rate their students’ classroom behaviors. 
The ESBA was developed by the study authors, and 
both intervention and comparison teachers received 
training in the use of the assessment. Study authors 
also examined impacts on two teacher outcomes, 
but those outcomes are not described in this WWC 
report because they were not eligible for review. 2 


WWC Rating 


The research described in this 
report meets WWC group design 
standards without reservations 

This study is a well-executed randomized controlled 
trial with low levels of sample attrition. 
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What did the study find? 

The study authors found, and the WWC confirmed, 
statistically significant and substantively important 
positive effects of WHS on the EBSA— a measure of 
student social skills. 


Features of We Have Skills! (WHS) 


WHS is a video-based social skills program for early 
elementary students. The intervention incorporates 
three components: instructional materials (including 
lesson plans, instructional videos, and supplemental 
activities), teacher professional development, and student 
assessment. This study focuses on the impact of the 
instructional and professional development components; 
teachers in both conditions received training in 
administering the student assessment component. 

Lessons included two main segments: a 5- to 10-minute 
discussion using questions from the lesson plan, and 
a 5-minute video lesson. The first video contains an 
introduction to the nature and importance of social 
skills, and seven subsequent videos cover different 
social skills, including guidance on how to: (a) listen, (b) 
ask for help, (c) follow directions, (d) do the best you 
can, (e) follow the rules, (f) work out strong feelings, 
and (g) get along. Instruction is delivered over eight 
weekly sessions; instruction in the current study began 
3 weeks after the start of the school year. Instruction 
is supplemented with guided practice activities and 
homework, along with reinforcement components. 

WHS teachers received training in administration of the 
ESBA assessment (the third intervention component) 
and 1 hour of additional training in administering the 
WHS intervention. They also received lesson plans and 
intervention materials for their classrooms. 
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Appendix A: Study details 

Marquez, B., Marquez, J., Vincent, C., Pennefather, J., Sprague, J., Smolkowski, K., & Yeaton, P. (2014). 
The iterative development and initial evaluation of We Have Skills!, an innovative approach to 
teaching social skills to elementary students. Education and Treatment of Children, 37(1), 137-161. 

Additional source: 

Pennefather, J., & Smolkowski, K. (2014). Validation of the Elementary Social Behavior Assess- 
ment: A measure of student prosocial school behaviors. Assessment for Effective Interven- 
tion. Advance online publication. doi:1 0.1 177/1 53450841 4557562 


Setting The study was conducted in 1 7 schools in four school districts, two in California and one each 
in Oregon and Washington. 

Study sample All teachers of grades K-3 in the four school districts were provided the opportunity to vol- 
unteer to participate in the study. After collecting teacher consent and teacher pretest data, 
classrooms within schools were randomly assigned to either the intervention group or the 
comparison group. In response to an author query, the authors indicated that a total of 83 
classrooms were initially randomized (46 to the intervention group and 37 to the comparison 
group), and a total of 70 classrooms were included in the study analysis (37 in the intervention 
group and 33 in the comparison group). Student-level baseline and analytic sample sizes were 
also obtained from the authors through author queries. A total of 1 ,661 students were initially 
present in the 70 study classrooms (890 in intervention classrooms and 771 in comparison 
classrooms), and 1 ,466 were included in the study analysis (822 in intervention classrooms 
and 644 in comparison classrooms). Of the 70 teachers in the analytic sample, 15 taught 
kindergarten, 23 taught first grade, 17 taught second grade, and 15 taught third grade. Demo- 
graphic data for students included in analytic sample were unavailable; instead, the authors 
report these data for all K-3 classrooms in study schools: an average of 42% of students were 
White; 29% were Latino; 12% were Asian/Pacific Islander; 7% were African American; 8% 
were multi-racial average, and 0.5% were American Indian/Alaskan; an average of 61 % of 
students were eligible for free or reduced-price lunch. 
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Intervention 

group 

WHS is a video-based social skills program for early elementary students. The intervention incor- 
porates three components: instructional materials (including lesson plans, instructional videos, 
and supplemental activities), teacher professional development, and student assessment. Les- 
sons included two main segments: a 5- to 10-minute discussion using questions from the lesson 
plan, and a 5-minute video lesson. The first video contains an introduction to the nature and 
importance of social skills, and seven subsequent videos cover different social skills, including 
guidance on how to: (a) listen, (b) ask for help, (c) follow directions, (d) do the best you can, (e) 
follow the rules, (f) work out strong feelings, and (g) get along. Instruction is delivered over eight 
weekly sessions; instruction in the current study began 3 weeks after the start of the school year. 

After the video and discussion segments were completed, teachers were encouraged to imple- 
ment activities such as guided practice through songs, role play, and learning materials. Materials 
included skills booklets and coloring pages, picture cards with feelings, and posters, some of which 
were supplementary or take-home materials. The program incorporates a reinforcement system 
where students receive skill tickets, student-managed tally sheets, parent notes of skill achieve- 
ment, and student certificates of mastery. Intensive “Tier II” interventions for struggling students, 
normally part of the intervention, were not implemented due to the short time frame of the study. 

The student assessment component of the intervention involves instruction with teachers on the 
use of the primary outcome measure, the Elementary School Behavior Assessment (ESBA), col- 
lected via the online irisPMT (Progress Monitoring Tool) interface developed by the authors/devel- 
opers. The teacher professional development component consists of a series of instructional videos 
and lesson plans that guide teachers’ implementation of the four types of student instructional 
materials: (a) video lessons, (b) practice opportunities, (c) songs, and (d) reinforcement systems. 

Of the intervention teachers, 34% reported spending 1 hour or less on the program; 16% 
spent 1-2 hours; 6% spent 2-3 hours; and 43% spent more than 3 hours during the 8-week 
duration of the study. 

Comparison 

group 

Teachers in the comparison group received training only in administering the ESBA via the 
irisPMT system; they received no training in the WHS intervention and were not asked to 
administer WHS in their classrooms. Comparison classrooms received access to the training 
and program at the completion of the evaluation. Twelve of the 1 7 schools participating in the 
study implemented School-Wide Positive Behavioral Interventions & Supports (SWPBIS); the 
authors do not present the proportion implementing SWPBIS by intervention status. 

Outcomes and 
measurement 

Student social skills were measured by the ESBA, which was completed by classroom teach- 
ers for each of their students. The assessment was administered before the intervention began 
and at end of the 8-week intervention. For a more detailed description of this outcome mea- 
sure, see Appendix B. 

Support for 
implementation 

After random assignment, teachers in both conditions participated in a training session on 
how to use the ESBA. The intervention teachers remained for an additional hour-long instruc- 
tion on the WHS curriculum. No additional support for implementation is described. 

Reason for 
review 

This study was identified for review by the WWC because it was supported by a grant to IRIS 
Media, Inc. (Principal Investigator: Brion Marquez) from the National Center for Special Educa- 
tion Research (NCSER) at the Institute of Education Sciences (IES). 
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Appendix B: Outcome measure for the social outcomes domain 


Social outcomes 


Elementary School Behavior Assessment The ESBA is a measure of student social skills. It is an online assessment administered through the irisPMT 
(ESBA) (Progress Monitoring Tool), a proprietary interface developed by the authors/developers. Teachers rate students on 

12 behavioral items that are designed to assess the behavioral skills that research has found teachers most prefer 
from the children in the classroom and that teachers agree are important for student success. The items cover 
topics such as listening, cooperating with others, following teacher directions, and working with effort. The ESBA 
uses a 3-point rating scale, including mastery, needs improvement, and cause for concern. Rating of a classroom 
is estimated to require 15-20 minutes. Internal reliability for the ESBA from the pretest and posttest showed 
Cronbach’s alpha = 0.95 and 0.94, with test-retest reliability = 0.77 (Pennefather & Smolkowski, 2014). 

Table Notes: Although the study was not directly eligible for review under an existing topic area protocol, the Children Classified as Having an Emotional Disturbance topic area 
review protocol (version 2.0) and the Teacher Training, Evaluation, and Compensation topic area review protocol (version 3.0) — related protocols — were both used to guide the 
determination of eligible outcomes and the appropriate outcome domain for the outcomes. The Children Classified as Having an Emotional Disturbance protocol was selected 
because it encompasses interventions targeted to develop social skills, the focus of the WHS intervention. The study was not directly eligible for review under this protocol because 
study participants were not classified or at risk of being classified as having an emotional disturbance. The Teacher Training, Evaluation, and Compensation protocol was also 
consulted for the two teacher outcomes: teacher self-efficacy (measured using the Teacher Sense of Efficacy Scale) and teacher satisfaction. Neither teacher outcome is eligible 
for review under either the Children Classified as Having an Emotional Disturbance protocol or the Teacher Training, Evaluation, and Compensation protocol, and therefore, are not 
included in this WWC report. 
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Appendix C: Study findings for the social outcomes domain 





Mean 

(standard deviation) 

WWC calculations 


Domain and 
outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Social outcomes 

Elementary School Behavior 
Assessment (ESBA ) — 

Total Score 

Grades 

K-3 

70 classrooms/ 
1,466 students 

33.61 

32.30 

1.31 

0.27 

+11 

< .01 

Domain average for social outcomes 





0.27 

+11 

Statistically 


significant 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on individual outcomes, representing the average change expected for all individu- 
als who are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the 
change in an average individual's percentile rank that can be expected if the individual is given the intervention. The statistical significance of the study's domain average was 
determined by the WWC. Some statistics may not sum as expected due to rounding. 

Study Notes: No corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-value presented here was reported in the 
original study. The authors analyzed program impacts using a multi-level ANCOVA model that controlled for pretest scores and accounted for the nesting of students within class- 
rooms. The study reports p-values and effect sizes from the multi-level ANCOVA model. Pretest and posttest means and standard deviations, as well as the intervention coefficient 
from the multi-level ANCOVA model, were provided by the authors in communications with the WWC. The WWC calculated the intervention group mean by adding the impact of the 
intervention, as estimated by the authors’ multi-level ANCOVA analysis, to the unadjusted comparison group posttest mean. The WWC effect size presented here is the same as 
the effect size reported by the authors in the original study. This study is characterized as having a statistically significant positive effect because the effect for the one measure 
within the domain is positive and statistically significant, and no effects are negative and statistically significant. For more information, please refer to the WWC Standards and 
Procedures Handbook (version 3.0), pp. 25-26. 
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Endnotes 

1 Single study reviews examine evidence published in a study (supplemented, if necessary, by information obtained directly from the 
authors) to assess whether the study design meets WWC group design standards. The review reports the WWC’s assessment of 
whether the study meets WWC group design standards and summarizes the study findings following WWC conventions for report- 
ing evidence on effectiveness. This study was reviewed using the single study review protocol, version 2.0. Two related protocols, the 
Children Classified as Having an Emotional Disturbance review protocol, version 2.0, and the Teacher Training, Evaluation, and Com- 
pensation review protocol, version 3.0, were consulted to guide the determination of eligible outcomes and to determine the appropri- 
ate outcome domain for the EBSA. The WWC rating applies only to the study outcomes that were eligible for review under these topic 
areas. The reported analyses in this SSR are only for those eligible outcomes that either met WWC group design standards without 
reservations or met WWC group design standards with reservations, and do not necessarily apply to all results presented in the study. 

2 See the table notes in Appendix B for more information. 

Recommended Citation 

U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. (2015, May). 

WWC review of the report: The iterative development and initial evaluation of We Have Skills!, an innovative 
approach to teaching social skills to elementary students. Retrieved from http://whatworks.ed.gov 
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Glossary of Terms 

Attrition 


Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Improvement index 


Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Single-case design 
(SCD) 

Standard deviation 


Statistical significance 
Substantively important 


Attrition occurs when an outcome variable is not available for all participants initially assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review if it falls within the scope of the review protocol and uses either 
an experimental or matched comparison group design. 

A demonstration that the analytic sample groups are similar on observed characteristics 
defined in the review area protocol. 

Along a percentile distribution of individuals, the improvement index represents the gain 
or loss of the average individual due to the intervention. As the average individual starts at 
the 50th percentile, the measure ranges from -50 to +50. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which study participants are 
assigned to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which eligible study participants are 
randomly assigned to intervention and comparison groups. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 

The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample are spread out over a large range of values. 

Statistical significance is the probability that the difference between groups is a result of 
chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% (p < .05). 

A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 


Please see the WWC Procedures and Standards Handbook (version 3.0) for additional details. 
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Intervention 

Report 



Practice 

Guide 



Quick 

Review 


Single Study 
Review 



A single study review of an individual study includes the WWC’s assessment of the quality of the research design 
and technical details about the study’s design and findings. 
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